CarND-Term1-P4¶
Advanced Lane Finding Project¶
The goals / steps of this project are the following:
- Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
- Apply a distortion correction to raw images.
- Use color transforms, gradients, etc., to create a thresholded binary image.
- Apply a perspective transform to rectify binary image ("birds-eye view").
- Detect lane pixels and fit to find the lane boundary.
- Determine the curvature of the lane and vehicle position with respect to center.
- Warp the detected lane boundaries back onto the original image.
- Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.
Processing Steps¶
Pre-steps¶
0 First -> Calibrate the camera, this to get the camera matrix (cMat) and distortion coefficients (coefs)
Main Pipeline steps:¶
- Undistort the image (distortion correction)
- Use color & gradient threshold
- Perspective transform (to view the image from top-down)
- Identify the lane lines in the warped image
- Back to the original perspective and draw lane and lines
Step 0 -> Camera calibration¶
Before any step, it's necessary get the camera distortion coeficients. We'll inpect pictures of a chessboard pattern using cv2 special functions. With this coefficients we'll be able to undistort the images.
In [5]:
#######################################################################################################################
import cv2
import numpy as np
import glob
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline
#######################################################################################################################
# CAMERA CALIBRATION (pre-step)
#######################################################################################################################
ret, cMat, coefs, rvects, tvects = None, None, None, None, None
def camera_calibration():
i = 0
im_paths = glob.glob('./camera_cal/calibration*.jpg')
cb_shape = (9, 6) # Corners we expect to be detected on the chessboard
obj_points = [] # 3D points in real-world space (undistorted)
img_points = [] # 2D points in the image
for im_path in im_paths:
img = mpimg.imread(im_path)
obj_p = np.zeros((cb_shape[0]*cb_shape[1], 3), np.float32)
coords = np.mgrid[0:cb_shape[0], 0:cb_shape[1]].T.reshape(-1, 2) # x, y coords
obj_p[:,:2] = coords
gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
found_all, corners = cv2.findChessboardCorners(gray, cb_shape, None)
if found_all:
img_points.append(corners)
obj_points.append(obj_p)
#img = cv2.drawChessboardCorners(img, cb_shape, corners, found_all)
else:
print("can't find all the corners in image:", im_path)
return cv2.calibrateCamera(obj_points, img_points, gray.shape, None, None)
#return: ret, mtx(camera matrix to transform 3d object to 2d image points), dist(dist coefs),
#rvecs(camera rotation vector), tvecs(camera translation vector)
ret, cMat, coefs, rvects, tvects = camera_calibration()
print("Camera distortion coefs:", coefs)
Plot functions¶
In [6]:
#######################################################################################################################
# Plot helper functions
#######################################################################################################################
# draw one big image (could fail because an unknown bug)
def one_plot(img):
# Plots one big image
fig, ax = plt.subplots(figsize=(20, 10))
ax.imshow(img)
# draw two medium images
def two_plots(img1, img2, img1_title=None, img2_title=None):
# Plots two images
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))
f.tight_layout()
ax1.imshow(img1)
if img1_title: ax1.set_title(img1_title, fontsize=30)
ax2.imshow(img2)
if img2_title: ax2.set_title(img2_title, fontsize=30)
Step 1 -> Undistort images¶
Left show the original image and right the undistorted image. The effect it's much easier to see in the chessboard image, but there are some signs of the effect in the image below (road), for example the white car on the bottom right.
In [7]:
#######################################################################################################################
# Step1: undistort images
#######################################################################################################################
def undistort_img(img):
if cMat is None: raise Exception('Error, camera is not calibrated -> exit')
return cv2.undistort(img, cMat, coefs, None, cMat)
original_image = mpimg.imread('./camera_cal/bad_imgs/calibration1.jpg')
img_undistorted = undistort_img(original_image)
# save image
cv2.imwrite('./output_images/undistorted_checkerboard.jpg', img_undistorted)
# nice plot
two_plots(original_image, img_undistorted, 'Original Img', 'Undistorted Img')
In [8]:
# testing undistort_img()
original_img = mpimg.imread('./test_images/test4.jpg')
undistorted_img = undistort_img(original_img)
two_plots(original_img, undistorted_img)
print('Testing undistort image')
Step 2 -> Use color && gradient threshold¶
It's needed a thresholding method that will utilize gradients and colors for lane lines detection in the images. Thresholding works better using a combination of the S-CHANNEL from HLS color space and Sobel Gradient.
In [14]:
#######################################################################################################################
# Step2: Use color & gradient threshold
#######################################################################################################################
def sobel_bin_img(img):
# require -> img should be 1-channel
# x-direction gradient
sobel = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=9)
abs_sobel = np.absolute(sobel)
scaled_sobel = np.uint8(255*abs_sobel/np.max(abs_sobel))
sobel_bin = np.zeros_like(scaled_sobel)
sobel_bin[(scaled_sobel >= 20) & (scaled_sobel <= 100)] = 1
return sobel_bin
def threshold_img(img, show=False):
# require -> im should be an undistorted image
# color-space conversion
gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
hls = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)
s_channel = hls[:,:,2]
# sobel gradient bins
sobel_s_bin = sobel_bin_img(s_channel)
sobel_gray_bin = sobel_bin_img(gray)
sobel_comb_bin = np.zeros_like(sobel_s_bin)
sobel_comb_bin[(sobel_s_bin == 1) | (sobel_gray_bin == 1)] = 1
# HLS S-Channel binary
s_bin = np.zeros_like(s_channel)
s_bin[(s_channel >= 150) & (s_channel <= 255)] = 1
comb_bin = np.zeros_like(sobel_comb_bin)
comb_bin[(sobel_comb_bin == 1) | (s_bin == 1)] = 1
gray_img = np.dstack((gray, gray, gray))
sobel_s_img = np.dstack((sobel_s_bin, sobel_s_bin, sobel_s_bin))*255
sobel_gray_img = np.dstack((sobel_gray_bin, sobel_gray_bin, sobel_gray_bin))*255
sobel_comb_img = np.dstack((sobel_comb_bin, sobel_comb_bin, sobel_comb_bin))*255
s_img = np.dstack((s_bin, s_bin, s_bin))*255
comb_img = np.dstack((comb_bin, comb_bin, comb_bin))*255
if show: two_plots(img, comb_img, 'Original', 'Thresholded')
return comb_img
In [15]:
# testing threshold (in 2 images)
original_img1 = mpimg.imread('./test_images/test2.jpg')
undistorted_img1 = undistort_img(original_img1)
threshold_img1 = threshold_img(undistorted_img1, show=True)
# save image
cv2.imwrite('./output_images/threshold_test2.jpg', threshold_img1)
original_img1 = mpimg.imread('./test_images/test4.jpg')
undistorted_img1 = undistort_img(original_img1)
threshold_img1 = threshold_img(undistorted_img1, show=True)
# save image
cv2.imwrite('./output_images/threshold_test4.jpg', threshold_img1)
print('Testing threshold')
Step 3 -> Perspective transform¶
With undistorted and thresholded images we get importante information, but also it's necessary limit that information by looking only at the portion of the image that is the only that matters (the road) To get that, we'll apply a perspective transform to shift our perspective to a top-down view of the road in front of the car.
In [16]:
#######################################################################################################################
# Step3: Perspective transform
#######################################################################################################################
def warp_to_lines(img, show=False):
# require ->img should be an undistorted image
x_shape, y_shape = img.shape[1], img.shape[0]
middle_x = x_shape//2
top_y = 2*y_shape//3
top_margin = 93
bottom_margin = 450
points = [
(middle_x-top_margin, top_y),
(middle_x+top_margin, top_y),
(middle_x+bottom_margin, y_shape),
(middle_x-bottom_margin, y_shape)
]
src = np.float32(points)
dst = np.float32([
(middle_x-bottom_margin, 0),
(middle_x+bottom_margin, 0),
(middle_x+bottom_margin, y_shape),
(middle_x-bottom_margin, y_shape)
])
M = cv2.getPerspectiveTransform(src, dst)
Minv = cv2.getPerspectiveTransform(dst, src)
warped = cv2.warpPerspective(img, M, (x_shape, y_shape), flags=cv2.INTER_LINEAR)
if show: two_plots(img, warped, 'Original', 'warped')
return warped, M, Minv
img = mpimg.imread('./test_images/straight_lines1.jpg')
img = undistort_img(img)
warped, M, Minv = warp_to_lines(img, show=True)
original_img = mpimg.imread('./test_images/test1.jpg')
undistorted_img = undistort_img(original_img)
thresholded_img = threshold_img(undistorted_img)
warped, M, Minv = warp_to_lines(thresholded_img, show=True)
cv2.imwrite('./output_images/warped_threshold_test2.jpg', warped)
Out[16]:
Step 4 -> Identify the lane lines in te warped image¶
We'll use a histogram at the bottom of the image to identify where the lanes start. The peaks should be our lane lines. Using a sliding-windows search we'll begin our search (using these peaks)
In [118]:
#######################################################################################################################
# Step4: Identify the lane lines in the warped image
#######################################################################################################################
warped_bin = np.zeros_like(warped[:,:,0])
warped_bin[(warped[:,:,0] > 0)] = 1
# Sum the columns in the bottom half of the image
histogram = np.sum(warped_bin[warped_bin.shape[0]//2:,:], axis=0)
plt.plot(histogram)
plt.savefig('./output_images/histogram.png')
In [18]:
def find_lane(warped, show=False):
# make binary version of the warped image
warped_bin = np.zeros_like(warped[:,:,0])
warped_bin[(warped[:,:,0] > 0)] = 1
# image we will draw on to show the lane-finding process
vis_img = warped.copy()
# max out non-black pixels so we can remove them later
vis_img[vis_img > 0] = 255
# sum the columns in the bottom portion of the image to create a histogram
histogram = np.sum(warped_bin[warped_bin.shape[0]//2:,:], axis=0)
# find the left an right right peaks of the histogram
midpoint = histogram.shape[0]//2
# x-position for the left window
left_x = np.argmax(histogram[:midpoint])
# x-position for the right window
right_x = np.argmax(histogram[midpoint:]) + midpoint
n_windows = 10
win_height = warped_bin.shape[0]//n_windows
# determines how wide the window is
margin = 80
pix_to_recenter = margin*2
# find the non-zero x and y indices
nonzero_ind = warped_bin.nonzero()
nonzero_y_ind = np.array(nonzero_ind[0])
nonzero_x_ind = np.array(nonzero_ind[1])
left_line_ind, right_line_ind = [], []
for win_i in range(n_windows):
win_y_low = warped_bin.shape[0] - (win_i+1)*win_height
win_y_high = warped_bin.shape[0] - (win_i)*win_height
win_x_left_low = max(0, left_x - margin)
win_x_left_high = left_x + margin
win_x_right_low = right_x - margin
win_x_right_high = min(warped_bin.shape[1]-1, right_x + margin)
###
# record the non-zero pixels within the windows
left_ind = (
(nonzero_y_ind >= win_y_low) &
(nonzero_y_ind <= win_y_high) &
(nonzero_x_ind >= win_x_left_low) &
(nonzero_x_ind <= win_x_left_high)
).nonzero()[0]
right_ind = (
(nonzero_y_ind >= win_y_low) &
(nonzero_y_ind <= win_y_high) &
(nonzero_x_ind >= win_x_right_low) &
(nonzero_x_ind <= win_x_right_high)
).nonzero()[0]
left_line_ind.append(left_ind)
right_line_ind.append(right_ind)
# if there are enough pixels, re-align the window
if len(left_ind) > pix_to_recenter:
left_x = int(np.mean(nonzero_x_ind[left_ind]))
if len(right_ind) > pix_to_recenter:
right_x = int(np.mean(nonzero_x_ind[right_ind]))
# combine the arrays of line indices
left_line_ind = np.concatenate(left_line_ind)
right_line_ind = np.concatenate(right_line_ind)
# gather the final line pixel positions
left_x = nonzero_x_ind[left_line_ind]
left_y = nonzero_y_ind[left_line_ind]
right_x = nonzero_x_ind[right_line_ind]
right_y = nonzero_y_ind[right_line_ind]
# color the lines on the vis_img
vis_img[left_y, left_x] = [254, 0, 0]
vis_img[right_y, right_x] = [0, 0, 254]
# fit a 2nd-order polynomial to the lines
left_fit = np.polyfit(left_y, left_x, 2)
right_fit = np.polyfit(right_y, right_x, 2)
# get our x/y vals for the fit lines
y_vals = np.linspace(0, warped_bin.shape[0]-1, warped_bin.shape[0])
left_x_vals = left_fit[0]*y_vals**2 + left_fit[1]*y_vals + left_fit[2]
right_x_vals = right_fit[0]*y_vals**2 + right_fit[1]*y_vals + right_fit[2]
if show:
fig, ax = plt.subplots(figsize=(20, 10))
ax.imshow(vis_img)
ax.plot(left_x_vals, y_vals, color='yellow')
ax.plot(right_x_vals, y_vals, color='yellow')
cv2.imwrite('./output_images/lane_detection_warped_test2.jpg', vis_img)
lane_lines_img = vis_img.copy()
# remove everything except the colored lane lines
lane_lines_img[lane_lines_img == 255] = 0
return y_vals, left_x_vals, right_x_vals, left_fit, right_fit, lane_lines_img
y_vals, left_x_vals, right_x_vals, left_fit, right_fit, lane_lines_img = find_lane(warped, show=True)
Step 5 -> Back to the original perspective and draw lane and lines¶
Now that we've identified the lane lines, with this information now we can draw over the original image the augmented lanes, applying a reverse perpective transform and drawing over the image. For that, we've created a basic lane line detection.
In [121]:
#######################################################################################################################
# Step5: Back to the original perspective and draw lane and lines
#######################################################################################################################
def draw_lane(img, lane_lines_img, y_vals, left_x_vals, right_x_vals, show=False):
# Prepare the x/y points for cv2.fillPoly()
left_points = np.array([np.vstack([left_x_vals, y_vals]).T])
right_points = np.array([np.flipud(np.vstack([right_x_vals, y_vals]).T)])
# right_points = np.array([np.vstack([right_x_vals, y_vals]).T])
points = np.hstack((left_points, right_points))
# Color the area between the lines (the lane)
lane = np.zeros_like(lane_lines_img) # Create a blank canvas to draw the lane on
cv2.fillPoly(lane, np.int_([points]), (0, 255, 0))
warped_lane_info = cv2.addWeighted(lane_lines_img, 1, lane, .3, 0)
unwarped_lane_info = cv2.warpPerspective(warped_lane_info, Minv, (img.shape[1], img.shape[0]))
drawn_img = cv2.addWeighted(img, 1, unwarped_lane_info, 1, 0)
if show:
#one_plot(drawn_img)
fig, ax = plt.subplots(figsize=(20, 10))
ax.imshow(drawn_img)
return drawn_img
img = original_img
#drawn_img = draw_lane(img, lane_lines_img, y_vals, left_x_vals, right_x_vals, show=True)
def basic_lane_detection_pipe(img, show=False):
if type(img) is str: img = mpimg.imread(img)
undist = undistort_img(img)
threshed = threshold_img(undist)
warped, M, Minv = warp_to_lines(threshed)
y_vals, left_x_vals, right_x_vals, left_fit, right_fit, lane_lines_img = find_lane(warped)
drawn_img = draw_lane(img, lane_lines_img, y_vals, left_x_vals, right_x_vals, show=show)
return drawn_img
print('Basic lane detection pipe')
drawn_img = basic_lane_detection_pipe('./test_images/test1.jpg', show=True)
drawn_img_wr = cv2.cvtColor(drawn_img, cv2.COLOR_BGR2RGB)
cv2.imwrite('./output_images/basic_lane_detection_pipeline.jpg', drawn_img_wr)
Out[121]:
Putting all together into a class -> Lane class¶
Determine how far rom center of lane the car is and estimates the curvature of the lane¶
To make the code more useful and efficient, we created a basic Lane class. Ass well, created the other method to meassuring the curvature of the lane and how far the car is from the center of the lane. (this information could be useful to calculate steering angles) The information HUD will be shown on the top-center of the image.
In [19]:
#######################################################################################################################
## Class Lane
#######################################################################################################################
class Lane():
def __init__(self):
self.y_vals = None
self.left_x_vals = None
self.right_x_vals = None
self.lane_lines_img = None
self.left_curvature = None
self.right_curvature = None
self.offset = None
def calc_curvature(y_to_fit, x_to_fit, y_eval):
# conversion factors for pixels to meters
m_per_pix_y, m_per_pix_x = 30/720, 3.7/700
# fit a new polynomial to world-space (in meters)
fit = np.polyfit(y_to_fit*m_per_pix_y, x_to_fit*m_per_pix_x, 2)
curvature = ((1 + (2*fit[0]*(y_eval*m_per_pix_y) + fit[1])**2)**1.5) / np.absolute(2*fit[0])
return curvature
def calc_offset(left_x, right_x, img_center_x):
lane_width = abs(left_x - right_x)
lane_center_x = (left_x + right_x)//2
pix_offset = img_center_x - lane_center_x
# expected the lane to be in meters
lane_width_m = 3.7
return lane_width_m * (pix_offset/lane_width)
def find_lane(warped, show=False):
# Create a binary version of the warped image
warped_bin = np.zeros_like(warped[:,:,0])
warped_bin[(warped[:,:,0] > 0)] = 1
vis_img = warped.copy() # The image we will draw on to show the lane-finding process
vis_img[vis_img > 0] = 255 # Max out non-black pixels so we can remove them later
# Sum the columns in the bottom portion of the image to create a histogram
histogram = np.sum(warped_bin[warped_bin.shape[0]//2:,:], axis=0)
# Find the left an right right peaks of the histogram
midpoint = histogram.shape[0]//2
left_x = np.argmax(histogram[:midpoint]) # x-position for the left window
right_x = np.argmax(histogram[midpoint:]) + midpoint # x-position for the right window
n_windows = 10
win_height = warped_bin.shape[0]//n_windows
margin = 50
pix_to_recenter = margin*2
# Find the non-zero x and y indexes
nonzero_ind = warped_bin.nonzero()
nonzero_y_ind = np.array(nonzero_ind[0])
nonzero_x_ind = np.array(nonzero_ind[1])
left_line_ind, right_line_ind = [], []
for win_i in range(n_windows):
win_y_low = warped_bin.shape[0] - (win_i+1)*win_height
win_y_high = warped_bin.shape[0] - (win_i)*win_height
win_x_left_low = max(0, left_x - margin)
win_x_left_high = left_x + margin
win_x_right_low = right_x - margin
win_x_right_high = min(warped_bin.shape[1]-1, right_x + margin)
###
# Record the non-zero pixels
left_ind = (
(nonzero_y_ind >= win_y_low) &
(nonzero_y_ind <= win_y_high) &
(nonzero_x_ind >= win_x_left_low) &
(nonzero_x_ind <= win_x_left_high)
).nonzero()[0]
right_ind = (
(nonzero_y_ind >= win_y_low) &
(nonzero_y_ind <= win_y_high) &
(nonzero_x_ind >= win_x_right_low) &
(nonzero_x_ind <= win_x_right_high)
).nonzero()[0]
left_line_ind.append(left_ind)
right_line_ind.append(right_ind)
# If there are enough pixels, re-align the window
if len(left_ind) > pix_to_recenter:
left_x = int(np.mean(nonzero_x_ind[left_ind]))
if len(right_ind) > pix_to_recenter:
right_x = int(np.mean(nonzero_x_ind[right_ind]))
# Combine the arrays of line indices
left_line_ind = np.concatenate(left_line_ind)
right_line_ind = np.concatenate(right_line_ind)
# Gather the final line pixel positions
left_x = nonzero_x_ind[left_line_ind]
left_y = nonzero_y_ind[left_line_ind]
right_x = nonzero_x_ind[right_line_ind]
right_y = nonzero_y_ind[right_line_ind]
# color the lines on the vis_img
vis_img[left_y, left_x] = [254, 0, 0] # 254 so we can isolate the white 255 later
vis_img[right_y, right_x] = [0, 0, 254] # 254 so we can isolate the white 255 later
# fit a 2nd-order polynomial to the lines
left_fit = np.polyfit(left_y, left_x, 2)
right_fit = np.polyfit(right_y, right_x, 2)
y_vals = np.linspace(0, warped_bin.shape[0]-1, warped_bin.shape[0])
left_x_vals = left_fit[0]*y_vals**2 + left_fit[1]*y_vals + left_fit[2]
right_x_vals = right_fit[0]*y_vals**2 + right_fit[1]*y_vals + right_fit[2]
# calculate real-world curvature for each lane line
left_curvature = calc_curvature(left_y, left_x, np.max(y_vals))
right_curvature = calc_curvature(right_y, right_x, np.max(y_vals))
offset = calc_offset(left_x_vals[-1], right_x_vals[-1], warped.shape[1]//2)
if show:
fig, ax = plt.subplots(figsize=(20, 10))
ax.imshow(vis_img)
ax.plot(left_x_vals, y_vals, color='yellow')
ax.plot(right_x_vals, y_vals, color='yellow')
lane_lines_img = vis_img.copy()
lane_lines_img[lane_lines_img == 255] = 0
lane = Lane()
lane.y_vals = y_vals
lane.left_x_vals = left_x_vals
lane.right_x_vals = right_x_vals
lane.lane_lines_img = lane_lines_img
lane.left_curvature = left_curvature
lane.right_curvature = right_curvature
lane.offset = offset
return lane
def draw_values(img, lane, show=False):
font = cv2.FONT_HERSHEY_DUPLEX
scale = .8
color = (0, 0, 0)
line_type = cv2.LINE_AA
cv2.putText(img,'LEFT/RIGHT Curvature: {:.2f}m / {:.2f}m'.format(lane.left_curvature, lane.right_curvature), (400, 40), font, scale, color, lineType=line_type)
cv2.putText(img,'CENTER-LANE: {:.2f}m'.format(lane.offset),(400, 80),font,scale,color,lineType=line_type)
if show:
fig, ax = plt.subplots(figsize=(20, 10))
ax.imshow(img)
return img
def draw_lane(img, lane, show=False):
left_points = np.array([np.vstack([lane.left_x_vals, lane.y_vals]).T])
right_points = np.array([np.flipud(np.vstack([lane.right_x_vals, lane.y_vals]).T)])
points = np.hstack((left_points, right_points))
filled_lane = np.zeros_like(lane.lane_lines_img)
cv2.fillPoly(filled_lane, np.int_([points]), (0, 255, 0))
warped_lane_info = cv2.addWeighted(lane.lane_lines_img, 1, filled_lane, .3, 0)
unwarped_lane_info = cv2.warpPerspective(warped_lane_info, Minv, (img.shape[1], img.shape[0]))
drawn_img = cv2.addWeighted(img, 1, unwarped_lane_info, 1, 0)
drawn_img = draw_values(drawn_img, lane)
if show:
fig, ax = plt.subplots(figsize=(20, 10))
ax.imshow(drawn_img)
return drawn_img
def detect_lane_pipe(img, show=False):
if type(img) is str: img = mpimg.imread(img)
undist = undistort_img(img)
threshed = threshold_img(undist)
warped, M, Minv = warp_to_lines(threshed)
lane = find_lane(warped)
return draw_lane(img, lane, show=show)
drawn_img = detect_lane_pipe('./test_images/test1.jpg', show=True)
drawn_img_wr = cv2.cvtColor(drawn_img, cv2.COLOR_BGR2RGB)
cv2.imwrite('./output_images/lane_detection_pipeline.jpg', drawn_img_wr)
Out[19]:
Tips and tricks¶
Filtering && smoothing¶
For filtering is necessary to look at previous Lane detections and determine whether the current Lane detection is appropiate (if current is too far of from the previous -> discard that information)
For smoothing we'll use the average of the current and previously saved searchs in that way our updates don't differ so mch from the previous result, this will produce a smoother video
In [20]:
#######################################################################################################################
## Tips && tricks
#######################################################################################################################
## Filtering
#######################################################################################################################
def filtering_lane(lane):
global prev_lanes
good_line_diff = True
good_lane_area = True
# ? the total x-pixel difference between the new and the old
if len(prev_lanes) > 0:
prev_x_left = prev_lanes[0].left_x_vals
prev_x_right = prev_lanes[0].right_x_vals
current_x_left = lane.left_x_vals
current_x_right = lane.right_x_vals
left_diff = np.sum(np.absolute(prev_x_left - current_x_left))
right_diff = np.sum(np.absolute(prev_x_right - current_x_right))
lane_pixel_margin = 50 # difference between new and old line
diff_threshold = lane_pixel_margin*len(prev_x_left)
if left_diff > diff_threshold or right_diff > diff_threshold:
print(diff_threshold, int(left_diff), int(right_diff))
#print()
good_line_diff = False
# fix the area between the lane lines (check if is appropriate )
lane_area = np.sum(np.absolute(np.subtract(lane.right_x_vals, lane.left_x_vals)))
area_min, area_max = 400000, 800000 # a^2 thresholds
if lane_area < area_min or lane_area > area_max:
good_lane_area = False
return (good_line_diff and good_lane_area)
#######################################################################################################################
## Smoothing
#######################################################################################################################
def smoothing_lane():
global prev_lanes
if len(prev_lanes) == 0: return None
elif len(prev_lanes) == 1: return prev_lanes[0]
else:
n_lanes = len(prev_lanes)
new_lane = prev_lanes[0]
avg_lane = Lane()
avg_lane.y_vals = new_lane.y_vals
avg_lane.lane_lines_img = new_lane.lane_lines_img
# avg L&R lanes -> x-values
left_avg = new_lane.left_x_vals
right_avg = new_lane.right_x_vals
for i in range(1, n_lanes):
left_avg = np.add(left_avg, prev_lanes[i].left_x_vals)
right_avg = np.add(right_avg, prev_lanes[i].right_x_vals)
avg_lane.left_x_vals = left_avg / n_lanes
avg_lane.right_x_vals = right_avg / n_lanes
# avg the curvatures and offsets
avg_lane.left_curvature = sum([lane.left_curvature for lane in prev_lanes])/n_lanes
avg_lane.right_curvature = sum([lane.right_curvature for lane in prev_lanes])/n_lanes
avg_lane.offset = sum([lane.offset for lane in prev_lanes])/n_lanes
return avg_lane
#######################################################################################################################
## Advanced lane detection pipe (with filtering && smoothing)
#######################################################################################################################
def advanced_lane_detection_pipe(img, show=False):
global prev_lanes
global n_bad_lanes
if type(img) is str: img = mpimg.imread(img)
n_lanes_to_keep = 5
if len(prev_lanes) == n_lanes_to_keep and n_bad_lanes == 0:
n_bad_lanes += 1
return draw_lane(img, smoothing_lane(), show=show)
undist = undistort_img(img)
threshed = threshold_img(undist)
warped, M, Minv = warp_to_lines(threshed)
lane = find_lane(warped)
if filtering_lane(lane):
n_bad_lanes = 0
prev_lanes.insert(0, lane)
if len(prev_lanes) > n_lanes_to_keep: prev_lanes.pop()
else:
n_bad_lanes += 1
# If we get stuck on some bad lanes, don't reinforce it, just clear them out.
if n_bad_lanes >= 12:
n_bad_lanes = 0
prev_lanes = []
# If we start with some bad lanes, this will just skip the drawing
if len(prev_lanes) == 0: return img
return draw_lane(img, smoothing_lane(), show=show)
# testing tips && trick (filtering && smoothing )
print('Testing filtering && smoothing')
n_bad_lanes = 0
prev_lanes = []
drawn_img = advanced_lane_detection_pipe('./test_images/test1.jpg', show=True)
cv2.imwrite('./output_images/filtering_smoothing_test.jpg', cv2.cvtColor(drawn_img, cv2.COLOR_RGB2BGR))
Out[20]:
Processing videos¶
It's time to test all with videos. There are 3 videos stored in directory 'test_videos', the output processed videos will stored in directory 'output_videos'.
In [21]:
#######################################################################################################################
from moviepy.editor import VideoFileClip
from IPython.display import HTML
# output_videos = {project_video.mp4, challenge_video.mp4, harder_challenge_video.mp4}
n_bad_lanes = 0
prev_lanes = []
video_name = 'project_video.mp4'
video = VideoFileClip('./test_videos/{}'.format(video_name))
video = video.fl_image(advanced_lane_detection_pipe)
%time video.write_videofile('./output_videos/{}'.format(video_name), audio=False)
In [136]:
n_bad_lanes = 0
prev_lanes = []
video_name = 'challenge_video.mp4'
video = VideoFileClip('./test_videos/{}'.format(video_name))
video = video.fl_image(advanced_lane_detection_pipe)
%time video.write_videofile('./output_videos/{}'.format(video_name), audio=False)
In [137]:
n_bad_lanes = 0
prev_lanes = []
video_name = 'harder_challenge_video.mp4'
video = VideoFileClip('./test_videos/{}'.format(video_name))
video = video.fl_image(advanced_lane_detection_pipe)
%time video.write_videofile('./output_videos/{}'.format(video_name), audio=False)
Results¶
With the 'project_video' the pipeline works very well.
With the 'challenge_video' the shadows on the underpass brigde make the lane detection very difficult.
With the 'harder_challenge_video' the pipeline breaks and get lost with the sharps curves and the shadows from the trees.